"From Elastic Net to Skynet"¶
What we will talk about¶
- Elastic net and other machine learning methods
- The general philosophy beyond machine learning-based analyses
What we will NOT talk about:¶
- "Skynet" from the Terminator movies
- Advanced machine learning models
- complex deep learning architectures
Sorry for clickbaiting you!
Aim of this talk / python notebook:¶
- Demonstrate the problem of overfitting
- Understand the difference between residual error and prediction error
- Introduction to regularization approaches and classical machine learning models
- Understand the need for training and test sets
- Understand the logic beyond cross validation and the black-box principle
- Outlook into some more elaborated machine learning models, like neural networks and deep learning
- Show a research example.
Python Notebook-based presentation¶
- figures are interactive
- source code is provided
- source code can be changed and run again on the fly
- feel free to interrupt the talk and patch the code if you have questions
Some technical details (not always shown in the slides).
In [97]:
# Imports and Co
import numpy as np
import pandas as pd
import plotly.io as pio
import plotly.figure_factory as ff
import plotly.offline as py #replaces import plotly.plotly as py
py.offline.init_notebook_mode()
import plotly.graph_objs as go
import plotly.express as px
from plotly.subplots import make_subplots
import teaching.teaching.plot as myplot
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import LeaveOneOut
from sklearn.preprocessing import StandardScaler
# Set default renderer
pio.renderers.default = 'notebook+jupyterlab' # See [1]
# Set default template
pio.templates['slides'] = go.layout.Template(layout=dict(width=800, height=550))
pio.templates.default = 'plotly+slides' # See [2]
In [114]:
def load_movie_data(start=0, end=100):
df_full = pd.read_csv("../data/movie_metadata.csv")
df_full['gross (log)']=np.log(df_full['gross'])
df_full['budget (log)']=np.log(df_full['budget'])
df_full = df_full.dropna()
df_full=df_full[['movie_title', 'title_year', 'budget', 'gross', 'imdb_score', 'duration', 'director_name', 'movie_facebook_likes',
'facenumber_in_poster', 'actor_1_facebook_likes', 'actor_1_name']]
return df_full[:100]
In [115]:
df = load_movie_data(0, 100)
df
Out[115]:
Can we predict the Gross income?¶
In [74]:
px.histogram(df, x="gross", height=400, marginal="rug")
In [117]:
df
Out[117]:
In [118]:
fig = px.scatter(df, x="budget", y="gross")
fig.update_layout(title_text='Budget $', height=400, width=600)
fig.show()
A possible "prediction": movie will have "zero balance".¶
In [119]:
fig = myplot.plot_scatter_with_regline(df, x="budget", y="gross", legend=('movie', 'zero balance'))
fig.show()
In [120]:
reg = LinearRegression().fit(X=df["budget"].values.reshape(-1,1), y=df["gross"])
In [121]:
print("Intercept:", reg.intercept_, "Slope:", reg.coef_)
print("Predicted income of a 1.000.000$ movie:")
reg.intercept_ + reg.coef_ * 1000000
Out[121]:
In [131]:
predicted = reg.intercept_ + reg.coef_*df["budget"]
#predicted = reg.predict(df["budget (log)"].values.reshape(-1,1))
fig = myplot.plot_scatter_with_regline(df, x="budget", y="gross",
predicted=predicted, legend=('movie', 'linear fit'))
fig.show()
Can we improve ourprediction by considering other features?¶
In [123]:
df
Out[123]:
In [124]:
px.scatter_3d(df, x="budget", y="gross", z="movie_facebook_likes")
Multiple Linear Regression¶
In [132]:
reg_multiple = LinearRegression().fit(X=df[['budget', 'duration', 'title_year', 'movie_facebook_likes']], y=df["gross"])
In [133]:
print("Intercept:", reg_multiple.intercept_)
print(' budget (log)', 'duration', ' title_year', ' movie_facebook_likes')
print("Slope:", reg_multiple.coef_)
print("Predicted income of a movie with:\n","budget=1.000.000$\n","duration: 120\n","year: 1996\n","likes: 100000\n",
"=", reg_multiple.predict(np.array([[1000000, 120, 1996, 100000]])), "$" )
In [134]:
predicted_multiple = reg_multiple.predict(df[['budget', 'duration', 'title_year', 'movie_facebook_likes']])
fig = myplot.plot_scatter_with_regpoints(df, x="budget", y="gross",
predicted=predicted_multiple, legend=('movie', 'linear fit'))
fig.show()
Predicted vs. Observed plots, prediction error¶
In [128]:
fig.show()
In [137]:
print("MSE:", mean_squared_error(df["gross"], predicted), mean_squared_error(df["gross"], predicted_multiple))
Prediction error improved as we added more features¶
- why not adding many features?
- Lets do an experiment: add many random features
In [140]:
def simulate(n_random):
return np.hstack((df['budget'].values.reshape(len(df['budget'].values),1), np.random.normal(0,1,(len(df['budget'].values), int(n_random)))))
mse=[]
for i, num_rand in enumerate(np.linspace(0, 79, 80)):
X=simulate(num_rand)
reg_random = LinearRegression().fit(X=X, y=df["gross"])
mse.append(mean_squared_error(df["gross"], reg_random.predict(X)))
if i%10 == 0: print(mse[i])
In [141]:
fig = px.scatter(x=np.linspace(0, 79, 80), y=mse, trendline="ols", labels={'x':'num random variables', 'y':'MSE'})
fig.show()
What on earth is happening?¶
- More explanatory variables -> more beta values to set
- A model with a sufficient number of (beta) parameters is able to "learn" the data at hand, instead of the relationship
BUT:
- Such a model will probably poorly generalize to new data (except: benign overfitting)
- Predictive power will be low
This is called:
Overfitting¶
Another example: adding ith degree of the variable (feature expansion)¶
Real examples of overfitting¶
Real examples of overfitting¶
RMS of models random variables on hold out data¶
In [143]:
df_train = df
df_test = load_movie_data(100,200)
def simulate(df, n_random):
return np.hstack((df['budget'].values.reshape(len(df),1), np.random.normal(0,1,(len(df['budget'].values), int(n_random)))))
mse_train, mse_test = [], []
for i, num_rand in enumerate(np.linspace(0, 79, 80)):
X_train=simulate(df_train, num_rand)
reg_random = LinearRegression().fit(X=X_train, y=df["gross"])
X_test=simulate(df_test, num_rand)
mse_test.append(mean_squared_error(df_test["gross"], reg_random.predict(X_test)))
if i%10 == 0: print(mse_test[i])
In [145]:
fig = make_subplots(rows=1, cols=2)
fig.add_trace(go.Scatter(x=np.linspace(0, 79, 80), y=mse, mode='markers', name="MSE on train"), row=1, col=1)
fig.add_trace(go.Scatter(x=np.linspace(0, 79, 80), y=mse_test, mode='markers', name="MSE on test"), row=1, col=2)
fig.update_layout(title_text='Adding random variables', height=300)
fig.show()
How can we know what is the optimal lambda?¶
In [146]:
loo=LeaveOneOut()
for train, test in loo.split(df):
print('train:', train)
print('test: ', test)
Ridge¶
In [149]:
X = simulate(df, 10)
y = StandardScaler().fit_transform(df["gross"].values.reshape(-1,1))
alphas = [1, 5, 50, 100]
mse=np.zeros((loo.get_n_splits(df), len(alphas)))
idx=0
for train, test in loo.split(df):
for i, alpha in enumerate(alphas):
ridge = Ridge(alpha=alpha).fit(X=X[train], y=y[train])
mse[idx, i] = mean_squared_error(ridge.predict(X[test]), y[test])
idx += 1
pd.DataFrame(mse, columns=alphas).mean()
Out[149]:
In [150]:
print(Ridge(alpha=1).fit(X=X, y=y).coef_.round(3).tolist())
print(Ridge(alpha=5).fit(X=X, y=y).coef_.round(3).tolist())
print(Ridge(alpha=50).fit(X=X, y=y).coef_.round(3).tolist())
print(Ridge(alpha=100).fit(X=X, y=y).coef_.round(3).tolist())
LASSO¶
In [151]:
np.random.seed(1)
X = simulate(df, 10)
y = StandardScaler().fit_transform(df["gross"].values.reshape(-1,1))
alphas = [0.0001, 0.001, 0.01, 0.1, 0.2]
mse=np.zeros((loo.get_n_splits(df), len(alphas)))
idx=0
for train, test in loo.split(df):
for i, alpha in enumerate(alphas):
lasso = Lasso(alpha=alpha).fit(X=X[train], y=y[train])
mse[idx, i] = mean_squared_error(lasso.predict(X[test]), y[test])
idx += 1
pd.DataFrame(mse, columns=alphas).mean()
Out[151]:
In [152]:
np.random.seed(1)
print(Lasso(alpha=0.0001).fit(X=X, y=y).coef_.tolist())
print(Lasso(alpha=0.001).fit(X=X, y=y).coef_.tolist())
print(Lasso(alpha=0.01).fit(X=X, y=y).coef_.tolist())
print(Lasso(alpha=0.1).fit(X=X, y=y).coef_.tolist())
print(Lasso(alpha=0.2).fit(X=X, y=y).coef_.tolist())
In [153]:
import tensorflow as tf
from tensorflow import keras
def preprocess(x, y):
x = tf.cast(x, tf.float32) / 255.0
y = tf.cast(y, tf.int64)
return x, y
def create_dataset(xs, ys, n_classes=10):
ys = tf.one_hot(ys, depth=n_classes)
return tf.data.Dataset.from_tensor_slices((xs, ys)) \
.map(preprocess) \
.shuffle(len(ys)) \
.batch(128)
# get data
(x_train, y_train), (x_val, y_val) = keras.datasets.fashion_mnist.load_data()
train_dataset = create_dataset(x_train, y_train)
val_dataset = create_dataset(x_val, y_val)
In [154]:
# set up
model = keras.Sequential([
keras.layers.Reshape(target_shape=(28 * 28,), input_shape=(28, 28)),
keras.layers.Dense(units=256, activation='relu'),
keras.layers.Dense(units=192, activation='relu'),
keras.layers.Dense(units=128, activation='relu'),
keras.layers.Dense(units=10, activation='softmax')
])
In [155]:
model.compile(optimizer='adam',
loss=tf.losses.CategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
history = model.fit(
train_dataset.repeat(),
epochs=5,
steps_per_epoch=200,
validation_data=val_dataset.repeat(),
validation_steps=2
)
